Skip to content

fix: #261 add support for other models that could produce JSON output in text content #532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

fix: #261 add support for other models that could produce JSON output in text content #532

wants to merge 1 commit into from

Conversation

@seratch
Copy link
Member

@seratch seratch commented Sep 26, 2025

This pull request resolves #261

Here are the actual request and response data in the scenario mentioned in the issue:

request

{
  "tools": [],
  "prompt": [
    {
      "role": "system",
      "content": "You are a helpful assistant\n\nYou must respond with a JSON object that matches this schema:\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"content\": {\n      \"type\": \"string\",\n      \"description\": \"Markdown formatted content \"\n    }\n  },\n  \"required\": [\n    \"content\"\n  ],\n  \"additionalProperties\": false,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\nReturn only valid JSON without code fences or extra commentary."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Write a haiku about  programming."
        }
      ],
      "providerOptions": {}
    }
  ],
  "responseFormat": {
    "type": "json",
    "name": "output",
    "schema": {
      "type": "object",
      "properties": {
        "content": {
          "type": "string",
          "description": "Markdown formatted content "
        }
      },
      "required": [
        "content"
      ],
      "additionalProperties": false,
      "$schema": "http://json-schema.org/draft-07/schema#"
    }
  }
}

response

{
  "responseId": "gen-1758930715-v132zfskAXPfAUDr5trQ",
  "usage": {
    "requests": 1,
    "inputTokens": 139,
    "outputTokens": 37,
    "totalTokens": 176,
    "inputTokensDetails": [],
    "outputTokensDetails": []
  },
  "output": [
    {
      "type": "message",
      "content": [
        {
          "type": "output_text",
          "text": "{\n  \"content\": \"Code flows like water,\\nBugs hide in silent shadows—\\nDebug, breathe, repeat.\"\n}"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "providerData": {
        "openrouter": {
          "provider": "Google",
          "usage": {
            "promptTokens": 139,
            "completionTokens": 37,
            "totalTokens": 176,
            "promptTokensDetails": {
              "cachedTokens": 0
            },
            "completionTokensDetails": {
              "reasoningTokens": 0
            },
            "costDetails": {
              "upstreamInferenceCost": 0
            }
          }
        }
      }
    }
  ],
  "providerData": {
    "content": [
      {
        "type": "text",
        "text": "{\n  \"content\": \"Code flows like water,\\nBugs hide in silent shadows—\\nDebug, breathe, repeat.\"\n}"
      }
    ],
    "finishReason": "stop",
    "usage": {
      "inputTokens": 139,
      "outputTokens": 37,
      "totalTokens": 176,
      "reasoningTokens": 0,
      "cachedInputTokens": 0
    },
    "warnings": [],
    "providerMetadata": {
      "openrouter": {
        "provider": "Google",
        "usage": {
          "promptTokens": 139,
          "completionTokens": 37,
          "totalTokens": 176,
          "promptTokensDetails": {
            "cachedTokens": 0
          },
          "completionTokensDetails": {
            "reasoningTokens": 0
          },
          "costDetails": {
            "upstreamInferenceCost": 0
          }
        }
      }
    },
    "request": {
      "body": {
        "model": "anthropic/claude-sonnet-4",
        "response_format": {
          "type": "json_schema",
          "json_schema": {
            "schema": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string",
                  "description": "Markdown formatted content "
                }
              },
              "required": [
                "content"
              ],
              "additionalProperties": false,
              "$schema": "http://json-schema.org/draft-07/schema#"
            },
            "strict": true,
            "name": "output"
          }
        },
        "messages": [
          {
            "role": "system",
            "content": "You are a helpful assistant\n\nYou must respond with a JSON object that matches this schema:\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"content\": {\n      \"type\": \"string\",\n      \"description\": \"Markdown formatted content \"\n    }\n  },\n  \"required\": [\n    \"content\"\n  ],\n  \"additionalProperties\": false,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\nReturn only valid JSON without code fences or extra commentary."
          },
          {
            "role": "user",
            "content": "Write a haiku about  programming."
          }
        ]
      }
    },
    "response": {
      "id": "gen-1758930715-v132zfskAXPfAUDr5trQ",
      "modelId": "anthropic/claude-sonnet-4",
      "headers": {
        "access-control-allow-origin": "*",
        "cf-ray": "9856ac076f8f986e-NRT",
        "connection": "keep-alive",
        "content-encoding": "gzip",
        "content-type": "application/json",
        "date": "Fri, 26 Sep 2025 23:51:56 GMT",
        "permissions-policy": "payment=(self \"https://checkout.stripe.com\" \"https://connect-js.stripe.com\" \"https://js.stripe.com\" \"https://*.js.stripe.com\" \"https://hooks.stripe.com\")",
        "referrer-policy": "no-referrer, strict-origin-when-cross-origin",
        "server": "cloudflare",
        "transfer-encoding": "chunked",
        "vary": "Accept-Encoding",
        "x-content-type-options": "nosniff"
      }
    }
  }
}

code

import { Agent } from '@openai/agents';
import { aisdk } from '@openai/agents-extensions';
import { createOpenRouter } from '@openrouter/ai-sdk-provider';
import { Runner } from '@openai/agents';
import { z } from 'zod';
import dotenv from 'dotenv';
dotenv.config();
const StartAgentResponseSchema2 = z.object({
  content: z.string().describe('Markdown formatted content ')
});

const modelName = 'anthropic/claude-sonnet-4';
const openrouter = createOpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY || ''
});
const model = aisdk(openrouter(modelName));

const runner = new Runner({
  model: model,
  tracingDisabled: true
});
const agent = new Agent({
  name: 'Assistant',
  instructions: 'You are a helpful assistant',
  outputType: StartAgentResponseSchema2
});

async function main() {
  try {
    const result = await runner.run(agent, 'Write a haiku about  programming.');
    console.log(result.finalOutput);
  } catch (e) {
    console.log(e);
  }
}

main().catch(console.error);

@changeset-bot
Copy link

changeset-bot bot commented Sep 26, 2025

🦋 Changeset detected

Latest commit: 9532dcc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@openai/agents-extensions Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@dkundel-openai
Copy link
Collaborator

dkundel-openai commented Sep 27, 2025

I'm not sure if this should be handled by the SDK? Shouldn't we just hand that off to the AI SDK and they enforce whatever is necessary to create structured output?

@seratch
Copy link
Member Author

seratch commented Sep 27, 2025

Yeah, I see your point. Currently the model responds this way:

Request:

{
  "tools": [],
  "prompt": [
    {
      "role": "system",
      "content": "You are a helpful assistant"
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Write a haiku about  programming."
        }
      ],
      "providerOptions": {}
    }
  ],
  "responseFormat": {
    "type": "json",
    "name": "output",
    "schema": {
      "type": "object",
      "properties": {
        "content": {
          "type": "string",
          "description": "Markdown formatted content "
        }
      },
      "required": [
        "content"
      ],
      "additionalProperties": false,
      "$schema": "http://json-schema.org/draft-07/schema#"
    }
  }
}

Response:

{
  "responseId": "gen-1758934002-qc2hRxL3IMtD9yZiwVfy",
  "usage": {
    "requests": 1,
    "inputTokens": 20,
    "outputTokens": 31,
    "totalTokens": 51,
    "inputTokensDetails": [],
    "outputTokensDetails": []
  },
  "output": [
    {
      "type": "message",
      "content": [
        {
          "type": "output_text",
          "text": "Here's a haiku about programming:\n\nCode flows like water—\nBugs hide in silent corners,\nLogic finds the way."
        }
      ],
      "role": "assistant",
      "status": "completed",
      "providerData": {
        "openrouter": {
          "provider": "Google",
          "usage": {
            "promptTokens": 20,
            "completionTokens": 31,
            "totalTokens": 51,
            "promptTokensDetails": {
              "cachedTokens": 0
            },
            "completionTokensDetails": {
              "reasoningTokens": 0
            },
            "costDetails": {
              "upstreamInferenceCost": 0
            }
          }
        }
      }
    }
  ],
  "providerData": {
    "content": [
      {
        "type": "text",
        "text": "Here's a haiku about programming:\n\nCode flows like water—\nBugs hide in silent corners,\nLogic finds the way."
      }
    ],
    "finishReason": "stop",
    "usage": {
      "inputTokens": 20,
      "outputTokens": 31,
      "totalTokens": 51,
      "reasoningTokens": 0,
      "cachedInputTokens": 0
    },
    "warnings": [],
    "providerMetadata": {
      "openrouter": {
        "provider": "Google",
        "usage": {
          "promptTokens": 20,
          "completionTokens": 31,
          "totalTokens": 51,
          "promptTokensDetails": {
            "cachedTokens": 0
          },
          "completionTokensDetails": {
            "reasoningTokens": 0
          },
          "costDetails": {
            "upstreamInferenceCost": 0
          }
        }
      }
    },
    "request": {
      "body": {
        "model": "anthropic/claude-sonnet-4",
        "response_format": {
          "type": "json_schema",
          "json_schema": {
            "schema": {
              "type": "object",
              "properties": {
                "content": {
                  "type": "string",
                  "description": "Markdown formatted content "
                }
              },
              "required": [
                "content"
              ],
              "additionalProperties": false,
              "$schema": "http://json-schema.org/draft-07/schema#"
            },
            "strict": true,
            "name": "output"
          }
        },
        "messages": [
          {
            "role": "system",
            "content": "You are a helpful assistant"
          },
          {
            "role": "user",
            "content": "Write a haiku about  programming."
          }
        ]
      }
    },
    "response": {
      "id": "gen-1758934002-qc2hRxL3IMtD9yZiwVfy",
      "modelId": "anthropic/claude-sonnet-4",
      "headers": {
        "access-control-allow-origin": "*",
        "cf-ray": "9856fc4e59386868-NRT",
        "connection": "keep-alive",
        "content-encoding": "gzip",
        "content-type": "application/json",
        "date": "Sat, 27 Sep 2025 00:46:45 GMT",
        "permissions-policy": "payment=(self \"https://checkout.stripe.com\" \"https://connect-js.stripe.com\" \"https://js.stripe.com\" \"https://*.js.stripe.com\" \"https://hooks.stripe.com\")",
        "referrer-policy": "no-referrer, strict-origin-when-cross-origin",
        "server": "cloudflare",
        "transfer-encoding": "chunked",
        "vary": "Accept-Encoding",
        "x-content-type-options": "nosniff"
      }
    }
  }
}

Indeed, the AI-SDK and the model should work nicely for this scenario. The code changes on this SDK side proposed in this PR is indeed a kind of workaround.

@seratch seratch marked this pull request as draft September 27, 2025 00:57
@seratch seratch closed this Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rm-openai rm-openai Awaiting requested review from rm-openai

@dkundel-openai dkundel-openai Awaiting requested review from dkundel-openai

Assignees

No one assigned

Labels

enhancement New feature or request package:agents-extensions

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ModelBehaviorError: Invalid output type when using @openrouter/ai-sdk-provider with outputType

3 participants

@seratch @dkundel-openai